only use nightly pytorch in ci #243
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
tushar00jain
force-pushed
the
pr243
branch
5 times, most recently
from
538ea8a to
dfe09bf
Compare
Summary: use http transport instead of pg transport -- pg transport fails to resolve address when running locally
This was referenced Jul 26, 2025
Merged
tushar00jain
force-pushed
the
pr243
branch
5 times, most recently
from
8017105 to
bfe221f
Compare
d4l3k
reviewed
Jul 28, 2025
torchft/manager.py
Outdated
| @@ -382,7 +382,7 @@ def allreduce(self, tensor: torch.Tensor, should_quantize: bool = False) -> Work | |||
| ) | |||
| else: | |||
| work = self._pg.allreduce([tensor], ReduceOp.SUM) | |||
| work.wait() | |||
| work.block_current_stream() | |||
There was a problem hiding this comment.
this partially solves it but it doesn't really help the case below with the tensor division
Ideally we wrap the future below in a Work object and then call .block_current_stream() on that
There was a problem hiding this comment.
this partially solves it but it doesn't really help the case below with the tensor division
in which case?
- for nccl with cuda, the behavior should be the same as the existing one
- for gloo with cuda, the tensor is on the gpu (after the host to device copy but the tensor arg is also on gpu) so the callback will also return immediately? iiuc the fut.wait() in the callback that i added will also return immediately
- for gloo without cuda, based on what you said the callback will be called after the device to host copy has been completed?
There was a problem hiding this comment.
Ideally we wrap the future below in a Work object and then call .block_current_stream() on that
we also need to call work.wait() or work.block_current_stream() to make sure work finishes on the current stream first before the future runs on the current stream
tushar00jain
force-pushed
the
pr243
branch
8 times, most recently
from
5507e47 to
61e177c
Compare
tushar00jain
force-pushed
the
pr243
branch
7 times, most recently
from
4698c70 to
d6b54f7
Compare
tushar00jain
changed the title
tushar00jain
force-pushed
the
pr243
branch
2 times, most recently
from
cb39d98 to
685e3c4
Compare
d4l3k
approved these changes
Jul 30, 2025
tushar00jain
force-pushed
the
pr243
branch
11 times, most recently
from
d650c7a to
09208a0
Compare
Summary: - call future.wait in callbacks to make sure the continuation executes after the future has completed - set the stream correctly to execute callback scheduled by bucketized allreduce
Summary: returns the work object so we can be more flexible with the usage
Summary: - change ci to only use nightly since block_current_stream is not in stable yet - fix new errors in nightly version of pyre - remove fixme[29] about future not being a function - make reduce_scatter_quantized return Work object
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Stack created with Sapling. Best reviewed with ReviewStack.